Time series based workload desc order optimization through reverse segment read #7244

gashutos · 2023-04-19T11:40:59Z

Description

EDIT -> After incorporating comments/suggestions from Nick, Andriya & Andorss, we decided to limit this change only to @timestamp field indices and also we changed default segment search order for these indices as Descending (from latest to the oldest segments), and also safe guarded rare ASC queries on time series based workload not to traverse on default DESC order to avoid any regression. The perf gains still remains similar.

As described in issue OpenSearch-6814, desc order sort queries are taking more time (3x or more) compare to asc order queries on same data set (time series based).
That is because all latest documents in time series workload are in latest segments when our IndexSearcher search documents in order of oldest segments to newest segments.

In this PR, we changed the leafReaderContext (segments) order in ContextIndexSearcher in reverse if query type is descending order sort. This will by default behaviour for all type of workloads and not just for time series based workloads. However it will only benefit to workload which has ever increasing data values. And for other type of workload, it wont regress or impact since data is randomly distributed across all segments.
But as a precaution, in case any user/workload gets search latency hit, I have added index level setting to disable this optimization.

Optimization gains

We are seeing 3x to 5x gains on below configuration.

On http_logs workload with disabled force merge, so each shard has around 100-150 segments.
instance type, r6g.2xlarge.
units are in ms.

Sort Type	hits = 5	hits = 100	hits = 1000
Desc	150	462	621
Desc (with this change)	49	112	125

Issues Resolved

OpenSearch-6814

Next action item

On same segment of time series based workload, ASC queiry is faster compare to descending order. We need to look further to that.

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: gashutos <[email protected]>

gashutos · 2023-04-19T11:46:04Z

@nknize @reta , check once you get time, if this appraoch looks good. The optimization gain is pretty good.

github-actions · 2023-04-19T12:01:53Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14293/
CommitID: a776ebb
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-04-19T12:09:08Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/14294/
CommitID: caed403
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

opensearch-trigger-bot · 2023-04-27T13:18:20Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7244-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 4c98b3d38064b94a87f6d7a1359e623849459bac
# Push it to GitHub
git push --set-upstream origin backport/backport-7244-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7244-to-2.x.

@timestamp

…gment read (opensearch-project#7244) This commit changes the EngineConfig for timeseries indexes only (e.g., indexes that use the @timestamp metadata field) so that a descending LeafSorter comparator is used to visit segments in order of most newest to oldest. For the more infrequent case that a user chooses to sort query results by ASC time, this would cause a search regression so the ContextIndexSearcher is updated to inspect the sort order from the search request and reverse the comparator so segments are visited in ascending order. LeafSorter behavior for non-timeseries indexes is left the same. Signed-off-by: gashutos <[email protected]> Signed-off-by: Chaitanya Gohel <[email protected]>

@timestamp

…gment read (opensearch-project#7244) This commit changes the EngineConfig for timeseries indexes only (e.g., indexes that use the @timestamp metadata field) so that a descending LeafSorter comparator is used to visit segments in order of most newest to oldest. For the more infrequent case that a user chooses to sort query results by ASC time, this would cause a search regression so the ContextIndexSearcher is updated to inspect the sort order from the search request and reverse the comparator so segments are visited in ascending order. LeafSorter behavior for non-timeseries indexes is left the same. Signed-off-by: gashutos <[email protected]> Signed-off-by: Chaitanya Gohel <[email protected]>

@timestamp

…gment read (#7244) (#7457) This commit changes the EngineConfig for timeseries indexes only (e.g., indexes that use the @timestamp metadata field) so that a descending LeafSorter comparator is used to visit segments in order of most newest to oldest. For the more infrequent case that a user chooses to sort query results by ASC time, this would cause a search regression so the ContextIndexSearcher is updated to inspect the sort order from the search request and reverse the comparator so segments are visited in ascending order. LeafSorter behavior for non-timeseries indexes is left the same. Signed-off-by: gashutos <[email protected]> Signed-off-by: Chaitanya Gohel <[email protected]>

…verse segment read (opensearch-project#7244)" This reverts commit 4c98b3d. Reverting due to issue reported in opensearch-project#7878.

…verse segment read (opensearch-project#7244)" This reverts commit 4c98b3d. Reverting due to issue reported in opensearch-project#7878. Signed-off-by: Andrew Ross <[email protected]>

…verse segment read (#7244)" (#7892) This reverts commit 4c98b3d. Reverting due to issue reported in #7878. Signed-off-by: Andrew Ross <[email protected]>

…verse segment read (#7244)" (#7892) This reverts commit 4c98b3d. Reverting due to issue reported in #7878. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit bb26536) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…verse segment read (#7244)" (#7893) This reverts commit 4c98b3d. Reverting due to issue reported in #7878. Signed-off-by: Andrew Ross <[email protected]>

…tion through re… (#7895) * Revert "Time series based workload desc order optimization through reverse segment read (#7244)" (#7892) This reverts commit 4c98b3d. Reverting due to issue reported in #7878. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit bb26536) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove unused imports Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <[email protected]>

…rough reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536.

…rough reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <[email protected]>

@timestamp

…gh reverse segment read (#7244)] with fixes (#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (#7244)" (#7892)" This reverts commit bb26536. Signed-off-by: gashutos <[email protected]> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <[email protected]> * Modifying CHANGELOG Signed-off-by: gashutos <[email protected]> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <[email protected]> --------- Signed-off-by: gashutos <[email protected]>

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <[email protected]> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <[email protected]> * Modifying CHANGELOG Signed-off-by: gashutos <[email protected]> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <[email protected]> --------- Signed-off-by: gashutos <[email protected]>

…gh reverse segment read (#7244)] with fixes (#7967) (#8037) Signed-off-by: gashutos <[email protected]>

…tion through re… (opensearch-project#7895) * Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892) This reverts commit 4c98b3d. Reverting due to issue reported in opensearch-project#7878. Signed-off-by: Andrew Ross <[email protected]> (cherry picked from commit bb26536) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * Remove unused imports Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andrew Ross <[email protected]>

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) (opensearch-project#8037) Signed-off-by: gashutos <[email protected]>

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <[email protected]> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <[email protected]> * Modifying CHANGELOG Signed-off-by: gashutos <[email protected]> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <[email protected]> --------- Signed-off-by: gashutos <[email protected]> Signed-off-by: Rishab Nahata <[email protected]>

@timestamp

…gment read (opensearch-project#7244) This commit changes the EngineConfig for timeseries indexes only (e.g., indexes that use the @timestamp metadata field) so that a descending LeafSorter comparator is used to visit segments in order of most newest to oldest. For the more infrequent case that a user chooses to sort query results by ASC time, this would cause a search regression so the ContextIndexSearcher is updated to inspect the sort order from the search request and reverse the comparator so segments are visited in ascending order. LeafSorter behavior for non-timeseries indexes is left the same. Signed-off-by: gashutos <[email protected]> Signed-off-by: Chaitanya Gohel <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

…verse segment read (opensearch-project#7244)" (opensearch-project#7892) This reverts commit 4c98b3d. Reverting due to issue reported in opensearch-project#7878. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <[email protected]> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <[email protected]> * Modifying CHANGELOG Signed-off-by: gashutos <[email protected]> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <[email protected]> --------- Signed-off-by: gashutos <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

gashutos added 5 commits April 19, 2023 02:38

Reverse segment search order in IndexSearcher for desc order queries

d9b936a

Signed-off-by: gashutos <[email protected]>

Adding more comments for code change reasing

f48aa3b

Signed-off-by: gashutos <[email protected]>

Adding cluster setting to disable/enable this optimization

e17faf0

Signed-off-by: gashutos <[email protected]>

Fixing spotlessApply

68376aa

Signed-off-by: gashutos <[email protected]>

Adding more null checks to be safe

a776ebb

Signed-off-by: gashutos <[email protected]>

gashutos requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and xuezhou25 as code owners April 19, 2023 11:41

Adding ChangeLog.md

caed403

Signed-off-by: gashutos <[email protected]>

reta added the backport 2.x Backport to 2.x branch label Apr 27, 2023

gashutos deleted the main-timeseries branch May 5, 2023 06:48

gashutos mentioned this pull request May 8, 2023

[Backport 2.x] Time series based workload desc order optimization through reverse se… #7457

Merged

Hailong-am mentioned this pull request Jun 2, 2023

[BUG] Reindex is break can not copy all documents from source #7878

Closed

andrross mentioned this pull request Jun 2, 2023

Revert "Time series based workload desc order optimization through re… #7892

Merged

6 tasks

gashutos added a commit to gashutos/OpenSearch that referenced this pull request Jun 8, 2023

Revert "Revert "Time series based workload desc order optimization th…

b5e29d0

…rough reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536.

gashutos mentioned this pull request Jun 8, 2023

Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

Merged

6 tasks

gbbafna pushed a commit that referenced this pull request Jun 13, 2023

Adding back [Time series based workload desc order optimization throu…

2b83949

…gh reverse segment read (#7244)] with fixes (#7967) (#8037) Signed-off-by: gashutos <[email protected]>

sohami mentioned this pull request Aug 24, 2023

[Concurrent Segment Search] For sort request on Timeseries index and field use non concurrent path #9538

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Time series based workload desc order optimization through reverse segment read #7244

Time series based workload desc order optimization through reverse segment read #7244

gashutos commented Apr 19, 2023 •

edited

Loading

gashutos commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

opensearch-trigger-bot bot commented Apr 27, 2023

Time series based workload desc order optimization through reverse segment read #7244

Time series based workload desc order optimization through reverse segment read #7244

Conversation

gashutos commented Apr 19, 2023 • edited Loading

Description

Optimization gains

Issues Resolved

Next action item

Check List

gashutos commented Apr 19, 2023

github-actions bot commented Apr 19, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Apr 19, 2023

Gradle Check (Jenkins) Run Completed with:

opensearch-trigger-bot bot commented Apr 27, 2023

gashutos commented Apr 19, 2023 •

edited

Loading